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Abstract: The assistive technology (AT) field 
acknowledges that different stakeholders vary 
in their perspectives of service delivery 
outcomes. While the literature delineates 
consumers, caregivers and providers as 
stakeholders with distinct views, very Utde 
research documents these unique 
perspectives. This study reports on the 
perspectives of (a) research-based federally 
funded, and (b) commercial AT product 
developers. Developers who received federal 
funding in 2001 were queried on their 
outcomes methodologies. Also, a random 
sample of manufacturers drawn from the 
ABLEDATA database and technology 
exhibitors at RESNA and AOTA conferences 
were surveyed. The data revealed that 
developers acknowledge the usefulness of 
outcomes data and relate that they would use 
outcomes information if available. 
Commercial manufacturers perceived cost as 
different from other outcome dimensions, 
interestingly, with lower importance. Also, 
formal research methods were used more 
frequendy than anticipated. This same group 
of commercial manufacturers also stated a gap 
between outcome measures they used and 
what they would use if available. This study 
contributes an important empirical snapshot 
of AT product developers and their 
perspectives of AT outcomes. 
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The Assistive Technology Outcome 
Measurement System (ATOMS Project) has 
undertaken a comprehensive needs 
assessment related to outcomes measurement 


and assistive technology (AT). This study 
surveyed the field to better understand how 
AT product developers and manufacturers 
view outcomes measurement. The field of AT 
outcomes has long recognized that there are 
various stakeholder views. DeRuyter (1998) 
stated that the clinical service delivery system 
must respond to several different 
performance monitoring dimensions: goals 
attainment/ results; functional status quality of 
Hfe; satisfaction; and cost. DeRuyter 
recognized that, “Each of these different 
dimensions in turn have varied significance to 
each of the different stakeholders, agencies, 
and sectors. While all stakeholders seek a 
successful outcome, not all stakeholders seek 
the same outcome” (p. 9). This discussion is 
followed with charts that delineate the 
perspectives of different stakeholders in terms 
of importance of various outcome dimensions 
and aspects of AT service. The 
“administrator,” “client,” “clinician,” and, 
“payer” (DeRuyter, p. 11-12) are on the Ust, 
but “manufacturers” or “product developers” 
are not. Lane (1997a) stated that, “while we 
often focus on assistive technology service 
delivery, there is a business context that is 
equally important” (p. 105). Addressing the 
needs for AT outcomes requires the 
consideration of what outcomes mean to 
product developers. This paper reports the 
results of a research effort to meet this need. 

Background 

The federal government funds technology- 
related research and development projects 
through multiple sources. The Small Business 
Innovative Research Program (SBIR), the 
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Small Business Technology Transfer Research 
Program (STTR), the NIDRR sponsored 
Rehabilitation Engineering Research Centers 
(RERC), and technology-related Research and 
Development (R&D) projects funded by the 
Office of Special Education Programs 
(OSEP) exist to promote state-of-the art 
development of AT. These investigators need 
to measure the success of their products 
under development. Similarly, private sector 
manufacturers and developers must do the 
same. In 1998, DeRuyter stated. 

Whether it is doing things right the 
first time or doing the right thing, 
accountability, performance 

monitoring, and the evaluation of 
outcomes has become the expected 
norm. While this has been embraced 
widely for many years by 
manufacturing, it needs to be fully 
embraced by the assistive technology 
community, (p. 8) 

Fuhrer (2001), however, suggests that 
developers stmggle to find appropriate 
outcomes instruments and methodologies for 
their products. Fuhrer, Jutai, Scherer, and 
DeRuyter (2003) list a variety of factors that 
may contribute to the shortfall of AT 
outcomes as compared to the growth of the 
AT industry. They comment that one of these 
factors is that there is a “greater emphasis of 
AT developers on demonstrating the technical 
performance of newly developed technology 
than on evaluating users’ performance with it” 
(p. 1244). The need is identified, but what 
actually is the state of outcomes measurement 
from the perspective of product 
development? There is a paucity of published 
work on this subject. 

The literature describes the importance of 
consumer input in product development and 
consumer evaluation during technology 
transfer to improve products. Multiple 
authors have discussed the importance of 


consumer input in the development of AT 
(Batavia & Hammer, 1990; Ryan, Rigby, & 
From, 1996; Vernardakis, Stephanidis, & 
Akoumianakis, 1994; Wessels, Willems, & de 
Witte, 1996). Compton (1995) states that 
many manufacturers test their concepts 
qualitatively to see how consumers perceive a 
potential new product before it is even 
produced. Lane (1998) describes the 
participatory action research approach of the 
Rehabilitation Engineering Research Center 
on Technology Evaluation and Transfer 
(RERC-TET) at the University of Buffalo. 
The Center involved individuals with 
disabilities in all aspects of its work, from 
grant development to program 
implementation. Consumers contributed to 
the evaluation of inventions, device 
commercialization, and the definition of ideal 
products. This last step resulted in a 
benchmarking process for AT product 
development. Benchmarking involves 
developing evaluation criteria using a 
methodology. Lane explains the value of this 
work. 

Manufacturers may use them [the 
benchmarks] to improve the product’s 
capabilities and gain the most return 
by focusing their design modifications 
in areas most important to the 
consumer. Vendors can use the 
benchmarks to emphasize desired 
attributes - and possibly down play 
undesirable attributes - when 
communicating their product’s value 
to customers, (p. 115) 

In discussion of the universal design process, 
Sanford, Story, and Ringholz (1998) also 
emphasize the importance of consumer 
inclusion. Such participation, they state, “has 
the potential to result in a number of 
outcomes that directly and indirectly benefit 
participants” (p. 161). 
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The European AT sector has also published 
discussions of the importance of user- 
centered design in AT. Poulson and 
Richardson (1998) describe the development 
of the USERfit design methodology in the 
UK: 

General consumer research has much 
to offer the assistive technology (AT) 
sector, and it is apparent that many 
AT companies now adopt a more 
marketing oriented approach to 
product development rather than a 
purely engineering perspective. AT 
companies are generally aware of the 
importance of understanding their 
customers wants and desires, but as 
yet many are not expert in obtaining 
such information from users or 
obtaining user feedback about the 
quality of their products (p. 163). 

They go on to state, “It is a common 
weakness of [product] design that litde 
emphasis is placed on evaluation activities, as 
it can be both difficult and expensive to carry 
out effectively” (p. 167). Consumer 

involvement in the design process, however, 
is not an outcome but a method by which it is 
hypothesized that better outcomes may be 
achieved. 

Similarly, the process of technology transfer 
provides some relevant discussion around the 
issue of outcome measures for AT product 
developers. Technology transfer is the process 
of taking product designs through the 
manufacturing process to maximize the 
success of the device in reaching the 
consumer. But again, this is a method directed 
towards improving outcomes. By itself, it does 
not produce outcome data. Lane (1997b) 
describes the sequence of the technology 
transfer process: (a) identification (of a 

technology and application); (b) research and 
development; (c) evaluation (testing with one 
or more clients); (d) transfer (of the 


technology to a buyer); and (e) 
commercialization. 

Evaluation of the product speaks to multiple 
aspects of AT outcomes. In an earlier 
publication. Lane, Usiak, and Moffat (1996) 
list consumer product evaluation criteria as: 
(a) reliability, (b) effectiveness, (c) physical 
comfort/ acceptability, (d) operability, (e) 
physical security/ safety, (1) durability, (g) 
learnabiUty, (h) portability, (i) securability, (j) 
maintenance/reparabiUty, and (k) 

affordability. Krass (1997) also writing on the 
issue of technology transfer, but from the 
perspective of one manufacturer, provides a 
detailed description of his company’s 
(Maddak, Inc.) two-step evaluation process: 

1 . Initial evaluation: 

2. Does the invention fit current product 
line? 

a. Product type 

b. Estimated retail price 

3. Is the product unique? 

4. Does it provide a clear benefit for 
users? 

5. Does product match Maddak’s 
manufacturing capabilities? 

6. In-depth evaluation: 

7. Who are the expected users? 

8. What is the market size? 

9. What is the competition? 

10. Is it safe to use? 

11. What is the manufacturing cost? 

12. Is tooling required and what is the 
cost of it? 

13. What is the acceptable retail price? 

14. Do the manufacturing cost and retail 
price match up? 

15. What is the estimated profit/year? 

16. Can tooling costs be paid for with two 
year’s profits? 

17. Is it estheticaUy pleasing? 

18. Is it patented? 

19. What is the “hunch” factor? 

20. What is the level of potential 
‘ownership’? (p. 57) 
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However, as Tobias (1997) states. 

As important as product development 
is to the business process, it is only 
one stage in a product’s life cycle and 
one element in its success. Product 
design lies within a constellation of 
activities such as market analysis, 
marketing, advertising and customer 
support, (p. 63) 

And so, as we look at the issue of outcomes 
measurement for the developers and 
manufacturers of AT, there is very litde 
information available. 

Many within the assistive technology 
community have developed the 
assumption that better outcomes are 
derived simply through improved 
technological solutions such as 
smaller, newer, faster, more portable 
and more sophisticated systems. 
Consequendy, technological soludons 
have been looked toward for 
improved outcomes without data to 
support the assumption. (DeRuyter, 
1997, p. 90) 

This investigadon attempted to idendfy gaps 
in the current state of outcomes measurement 
instruments and systems in the area of AT 
product development targedng both federally 
funded and private sector development. 

Research Questions 

Six primary questions directed 
the framework for this study. One general 
question focused on federal research projects: 
(Al) What methods for collecting AT 
outcomes data do currently funded federal 
projects project to use? Five questions 
focused on commercial product developers: 
(Bl) What importance do product developers 
place on outcome dimensions of AT? (B2) 
How frequently do product developers use 


specific strategies to measure outcome during 
development? (B3) How frequently do 
product developers use specific types of 
formal instrumentation to quantify outcome? 
(B4) How do product developers perceive the 
appropriateness of different types of 
standardized instrumentation? (B5) How 
would product developers use valid outcome 
data? 

Methods 

Sample 

Two samples were tapped to cover the two 
question domain areas. 

Sample (A): Federally funded projects. Two 
sampling methods identified federally funded 
projects. First, we examined the Computer 
Retrieval of Information on Scientific Projects 
(CRISP) database maintained by the Office of 
Extramural Research at the National Institute 
of Health (n.d.) on “assistive” and “assistive 
technology.” The search selected 32 
appropriate projects. Second, the NIDRR 
Program directory web page (National 
Rehabilitation Information Center, n.d.) listed 
61 records for their category “Research 
Priority: Technology for Access and 

Function.” A review of these abstracts 
identified 24 appropriate projects. In aU, 56 
projects were identified (with 3 researchers 
having two funded projects each) to make up 
the federally funded product developers 
group. 

Sample (B): Commercial product developers. Two 
methods were used to identify the commercial 
product developer group. The first method 
randomly sampled the “Directory of 
Manufacturers and Distributors” available on 
May, 2002, at the ABLEDATA (n.d.) website 
that provided a population of commercial 
product developers. It contained more than 
2,500 listings. Prior to random selection, the 
list was limited to companies in the U.S. that 
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were coded as (a) active, and (b) 
manufacturers. They totaled 1,124. From this, 
a statistical analysis software program 
generated a random sample of 500. The 
second method identified all U.S. based 
technology exhibitors from the RESNA 2001 
Conference {n = 33) and the American 
Occupational Therapy Association (AOTA) 
2001 Conference {n = 35). Some 

manufacturers exhibited at both conferences. 
The random sample {n = 500) was 

crosschecked with the exhibitor lists {n = 68), 
removing duplicates, for a final set of 555 
companies. 

Procedures 

Procedure (A): Federally funded projects. Due to 
the proprietary nature of much of the content 
of grant proposals, only the abstracts were 
available as public information. To obtain the 
necessary information, the project sent the 53 
identified principal investigators a letter 


explaining the purpose of the study and 


requesting the methodology to test the 
outcomes of their product(s) under 
development that was submitted with their 
proposal. 

Procedure (B): Commercial product developers. 
The survey sent to commercial product 
developers (see Appendix) was drafted based 
on findings from the AT service provider and 
consumer/user focus groups (Taugher, 2004) 
and suggested by the literature set, some of 
which is cited in this paper. 

Results 

Descriptive analyses of results are presented 
separately for each of the two groups 
surveyed as the groups and survey 

methodology differed. 

{A) Federally funded projects. Overall, the 
federal project survey obtained a 50% 
response rate. (Four letters were returned as 


undeliverable.) Questions were coded to 


Table 1 

Percentage of Methodology and Instrumentation Choices for 26 Funded Projects 


Strateg\r/Method % of uses 


Strategies for measuring outcome during product development 

Testing in lab by developers 27% 

Focus groups of providers 12% 

Focus Groups of persons with disabikties 15% 

User usability testing in lab 23% 

General field testing soliciting feedback from persons with disabilities 19% 

General field testing soliciting feedback from providers 12% 

Formal research design: Single Subject design 35% 

Formal research design: Group comparison 58% 

Other 0 

Use of formal instrumentation 

Standardized, valid measure of functional status 70% 

A “homemade” or adapted measure of improved functional performance 62% 
A measure of client satisfaction 23% 

A measure of cost 15% 

Other 0 
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correspond to the strategies for measuring 
outcome and types of measures, as were 
presented in the commercial developers’ 
survey (see Appendix, Question #3 and 
reported in Table 1). Due to the number of 
variables and complexity of the development 
process it was not uncommon for a reviewed 
project to cite more than one method or type 
of instrumentation. Nine different strategies 
or methods were identified as being used by 
these developers, with formal group designs 
being the most frequent, occurring in 58% of 
the projects. Two methods that sampled 
providers rather than consumers were the 
least frequently occurring, at 12% each. 
Regarding formal instrumentation, the use of 
standardized measures (used in 70% of the 
studies) slightly eclipsed the use of 
“homemade” or adapted measures (62%). No 
respondents reported themselves as using 
“other” instmmentation. This demonstrates 
an element of the validity of these categories 
established during the ATOMS service 
provider focus group process. See Table 1 for 
the distribution of responses. 

(B) Commercial product developers. Of the 555 
mailed surveys, 135 were returned as 
undeliverable. Of the remaining 420 surveys, 
10 individuals responded that they were no 
longer involved in production of AT devices 
or that they did not wish to participate. A 
total of 40 competed surveys were returned. 
The overall response rate was 12%. 


Interestingly, in 2003, the U.S. Department of 
Commerce Technology Assessment of the 
U.S. Assistive Technology Industry received a 
similar response rate. It mailed 1,600 surveys 
and received only 232 responses, or 14.5% 
initially. 

This low response rate was perplexing, but 
careful scmtiny provides a plausible 
explanation. It appeared that two different 
types of companies were reached through our 
sampling efforts. Indeed, the ABLEDATA 
(n.d.) Directory of Manufacturers and 
Distributors contains many companies who 
do not specialize in adaptive equipment (3M 
Co., Kohler Co., L.L. Bean, and Union 
Carbide Corporation, to list a few). It would 
not be surprising that these companies would 
not be motivated to respond to a survey about 
AT outcomes. Could we distinguish disability- 
focused companies and if we could, would 
that help explain the low response rate? 

We first examined the origin of the 40 
completed surveys to determine if they came 
from companies that were identified from the 
random sample of ABLEDATA (n.d.) 
companies or if they came from the 
specifically selected conference exhibitors. In 
fact, 38% {n — 15) of the 40 completed 
surveys came from companies that had 
exhibited at either the AOTA or RESNA 
annual conferences in 2001. 


Table 2 

Percent of Survey Response by T 5 rpe of Company 


Company Tj'pe 

Responded 

No Response 

Disability Manufacturers 

38% 

62% 

Non-Disability Manufacturers 

9% 

91% 

Total 

12% 

88% 
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Figure 1. Distribution of product developers’ responses for importance of AT outcomes. Six of 
seven categories show distributions similar to la. The importance of Cost (lb) demonstrates the 
seventh, a bi-modal distribution. 


1a: Importance of Change in Performance, Usage, 
Satisfaction, Increased life Participation, Improved Quality 
of Life, & Goai Achievement as AT Outcomes 



1 b: Importance of Cost as an AT Outcome 



Secondly, we wondered if any companies in 
our sample, the 555 mailed surveys, were 
members of the Assistive Technology 
Industry Association (ATIA). We retrieved 
the membership list from the ATIA Web site 
to see if any of the companies in our sample 
were members. From the original mailing of 
555, 15 of the surveys were sent to companies 


that belonged to ATIA. Two of these were 
returned as undeliverable. Five surveys (38%) 
were returned from the remaining 13 
companies. 

We then combined the identified exhibitor 
companies and the identified ATIA 
companies, removing duplicates, to form the 
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Table 3 

Mean Responses to Importance of Specific AT Outcomes, Product Developers 


Category 

M 

Change in performance or function 

4.5 

Cost 

3.7 

Usage: Why or why not used 

4.2 

Consumer satisfaction 

4.6 

Increased life participation 

4.2 

Improved quality of life 

4.4 

Clinical result/ goal achievement 

4.1 


new category “disability manufacturers.” The performed on the survey responses with SPSS 

remaining companies became “non-disability (Statistical Package for the Social Sciences) 

manufacturers.” See Table 2 for percentages version 11.5 for Windows. Data are discussed 

of response rate for these two categories. in order of the research questions delineated 

Clearly, it appears that companies that are earlier, 

actively involved in marketing to professionals 

in the AT service delivery system and who are /. What importance do commercial product 

active in developing AT are more likely to developers place on outcome dimensions of AT? 

take part in research regarding AT outcomes. Respondents were asked to rate seven 

outcome dimensions of AT on a scale of “not 
Descriptive statistics, as appropriate, were at aU important” to “extremely important.” 


Table 4 

Frequency of Methodology Use, Product Developers 


Methodology 

Not at all 

Less than half of 
the time 

More than half of 
the time 

Testing in lab by developers 

12.5% 

12.5% 

75% 

Focus groups of providers 

25% 

30% 

44% 

Focus groups of persons with 
disabilities 

17.5% 

32.5% 

48% 

User usability testing in lab 
General field testing soliciting 

22.5% 

22.5% 

55% 

feedback from persons with 
disabilifies 

10% 

15% 

75% 

General field testing solicidng 
feedback from providers 

20% 

17.5% 

62% 

Formal research: Single-subject 
design 

45% 

15% 

35% 

Formal research: Group comparison 
design 

45% 

10% 

37% 

Other 

0% 

0% 

0% 
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Table 5 

Frequency of Use of Formal Instrumentation During Product Development for Commercial 
Product Developers 


Instmmentation 

Did not use 

Less than half 
of the time 

More 
half of 
time 

than 

the 

Total 

Standardized measure of 
functional performance 

47.5% 

12.5% 

40% 


100% 

A “homemade” or adapted 
measure of improved 
functional performance 

35% 

15% 

50% 


100% 

A measure of client satisfaction 

25% 

7.5% 

67.5% 


100% 

A measure of cost 

20% 

22.5% 

57.5% 


100% 

Other 

0% 

0% 

0% 


0% 


Again, the seven categories evolved from the 
focus group process mentioned earlier, and 
correlate with DeRuyter’s (1998) categories. 
The categories were: (a) change in 

performance of function, (b) cost, (c) usage- 
why or why not used, (d) consumer 
satisfaction, (e) increased life participation, (1) 
improved quality of Hfe, and (g) result/goal 
achievement. The histogram of response to 
the category “Increased Life Participation” is 
shown in Figure la. It demonstrates the 
similar pattern of response that was observed 
for six of the seven categories. Cost, however, 
demonstrated a significandy different pattern 
with a bi-modal distribudon, as demonstrated 
in Figure lb. 

Table 3 lists the mean responses to all 
categories in this question. While the mean 
score for the importance of cost is sUghtiy 
lower than for the other categories, the bi- 
modal distribution of this variable suggests 
that the importance of cost as an outcome is 
important, but not for everybody. 


developers for their use of specific strategies. 
Formal research designs are the least 
frequently employed strategies for the 
commercial developers group. The results are 
not surprising as commercial companies may 
not have research-trained staff in-house. 
Consultation is costly. While it is difficult to 
compare the results of the open needed 
question responses of the federally funded 
researchers with the Likert-Uke scale 
responses from the survey, the Pis reported 
using single- subject designs 36% of the time 
and group comparison designs 58% of the 
time (see Table 1). Interesting, however, was 
one comment from a respondent from the 
federally funded group. That researcher, 
receiving SBIR funding, complained about 
how difficult it was to set up a solid research 
design to meet the requirements of the grant. 
She felt strongly that there are not adequate 
resources available for small businesses to 
achieve consulting services at reasonable 
costs. 


2. How frequently do commercial product 
developers use specific strategies to measure outcome 
during development? Table 4 lists the frequency 
reported by the commercial product 


3. How frequently do product developers use 
specific types of formal instrumentation to quantify 
outcome? Table 5 contains the aggregated 
response data from the question, “If you have 
used formal instmmentation as a form of 
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quantifying outcome during product 
development, how frequently have you 
used...?” for those who responded that they 
used formal instrumentation. 

Not surprisingly, commercial manufacturers 
infrequendy use standardized measures. This 
is compared to a 70% use rate for the 
federally funded projects (see Table 1). 
Clearly, client satisfaction dominated as the 
type of instrumentation used by the 
commercial developers group. 

Returning to the issue of cost, it is interesting 
that cost is reported as being used so 
frequently, despite it’s low ranking on the 
importance dimension (commercial product 
developer research question #1, above). 
Considering that the survey question asks 
about use of formal instrumentation, it could 
be argued that cost data is much easier for 
commercial manufacturers to obtain. 

4. How do product developers perceive the 
appropriateness of different types of standardised 
instrumentation? Figure 2 shows the commercial 
product developers’ responses to the 


question, “If standardized instmmentation 
were available for each of the following 
outcomes. How appropriate would each of 
the following be for your product 
development? 

Commercial product developers felt that self- 
satisfaction measures, cost measures, 
functional performance measures and focus 
group protocols would all be appropriate 
measures for them if standardized 
instrumentation were available. The mail and 
survey measures were not considered as 
useful. These findings reflect the “hands-on” 
perspective of manufacturers with less of a 
mandate for longer-term follow-up, 
presumably the function of mail and 
telephone surveys. Figure 3 shows the rank 
ordering of the categories when focusing only 
on those who responded “always” for 
potential use of each of the types of 
standardized instrumentation. It appears that 
standardized instrumentation for change in 
functional performance would be useful for 
product developers. 

5. How would product developers use valid 


Figure 2. Appropriateness of each type of standardized instrumentation, if available, for 
commercial product developers 


A. Self Satisfaction Measure 




1 


1 


1 

2 

n m 




Wouldn't Frequently Always 


Appropriateness 


C. Functional Performance Impact Measure 



Wouldn't Frequently Always 


Appropriateness 


B. Cost Measure 



Appropriateness 


D. Focus Group Protocol 



Appropriateness 
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Figure 3. Number of product developers responding “always” (giving a score of 5) for potential 
use of each type of standardized instrumentation if available 


Number of "always" responses 

0 2 4 6 8 10 12 14 16 

Functional performance impact measure 

Cost 

Self satisfaction measure 
Focus group protocol 
Mail Survey measure 
Telephone survey measure 



outcome data? Figure 4 shows the distribution 
of responses to the question, “If you had vaUd 
outcome data about your products, how likely 
would you be to use it for the following 
business purposes?” The lines on each graph 
separate the responses and form two general 
categories, “wouldn’t be used much” and 
“would be used a lot.” 

This shows that if vaUd outcomes data were 
available, commercial product developers 
would most Ukely use it for advertising and 
product development. It does not appear that 
there is a strong interest by this group to use it 
for funding or monies acquisition. 

Discussion, Outcomes, and Benefits 

This study investigated the current use of 
outcomes measures by two groups of AT 
developers, those who received federal funds 
for their development projects and those who 
developed AT within their businesses without 
federal support. The analysis of federaUy 
funded investigators was based solely on what 
they defined in the evaluation plans of their 
federal grant proposals that received funding. 


Commercial developers of AT provider 
specific survey data on their use and 
perception of outcomes measures. Perhaps 
one of the most surprising findings was the 
response of the commercial product 
developers to the importance of cost as an AT 
outcome dimension. While cost would seem 
to be overt and prominent in a business 
setting, the descriptive data demonstrates that 
commercial manufacturers saw cost as 
different from the other outcome dimensions. 
Its bimodal distribution demonstrates a lower 
importance for cost as an outcome. Also, the 
reported use of formal research design 
methods and data collection was more 
prominent than what we thought to be the 
case for “inventors.” Finally, for the 
commercial respondents there is a gap 
between what outcomes measures they would 
use, if available, and what they currently use. 
This appears to be a technology transfer 
problem. Outcome instruments do not appear 
to be making their way to the commercial 
sector. Maybe this reflects the relatively young 
age of outcomes instmmentation and that 
many outcomes instmments remain in the 
research and development phase. 
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Figure 4. Commercial product developers likelihood of use of valid outcomes data if available 



C. Product Revision/Improvement During Development 
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Table 6 

Comparison of Objective (Funded Proposals) and Subjective (Commercial Developers) Data 



Objective of 

Subjective Report 

Strategy/Method 

Use' 

of Use^ 


% 

% 

Strategies for measuring outcome during 
product development 
Testing in lab by developers 

27% 

87.5% 

Focus groups of providers 

12% 

72.5% 

Focus groups of persons with disabilities 

15% 

80% 

User usability testing in lab 

23% 

77.5% 

General field testing soliciting feedback from 
providers 

19% 

90% 

General field testing soliciting feedback from 
providers 

12% 

77.5% 

Formal research design: Single subject design 

35% 

47.5% 

Formal research design: Group comparison 

58% 

42.5% 

Other 

0 

0 

Use of formal instmmentation 



Standardized, valid measure of functional status 
A “homemade” or adapted measure of improved 
functional performance 

70% 

62% 

52.5% 

65% 

A measure of client satisfaction 

23% 

75% 

A measure of cost 

15% 

80% 

Other 

0 

0 


'Reported in federally funded proposals 
^ Commercial product developers survey 


A post hoc comparative analysis of the data 
from funded R&D projects and commercial 
developers reveals interesting differences, but 
must be read with caution. Two different 
methods were used to gather the information 
in this project. The federally funded 
developers submitted the actual “methods” 
section from their grant proposals. We have 
no evidence that these are the methods they 
actually used as their projects evolved; it 
speaks only to their planned evaluation 
process intent. The commercial product 
developers, however, responded to a survey 
with Likert-like response scales, thus 
providing a subjective response to the 
questions as they have dealt with the 
questions in their businesses over time. 
Accepting these differences, we created Table 
6. Data from the federally funded proposals 


are exactly as they appear in Table 1. To 
obtain the numbers for the commercial 
developers, we subtracted the percentage of 
“not at aU” responses (Tables 3 and 4) from 
100% to obtain the numbers in the right hand 
column of Table 6. Some interesting 
comparisons appear. Clearly, during product 
development for the federally funded 
researchers, there is a dearth of consumer or 
service provider input. This represents a rich 
source not being tapped without input from 
AT device users. Also, the instmmentation 
data for this group demonstrates a proclivity 
for performance, a somewhat myopic 
perspective of outcome. While these 
comparisons are presented for discussion 
purposed only, they are a reminder of the fact 
that a comprehensive model of AT outcomes 
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information must consider both objective and 
subjective data. 

A limitation of this study was the low 
response rate for the commercial product 
developers. Compared to federally funded 
projects, commercial developers did not use 
standardized methodology or instmmentation 
as frequendy. However, they did express an 
interest in outcomes. The repordng of the use 
of homemade measures of functional 
performance and client satisfaction measures 
represent their current efforts to keep the 
consumer in mind as they advance their 
technological solutions. They acknowledge 
that standardized outcomes data would be 
useful and relate that they would use 
standardized outcomes data if it was available. 

This investigation is an important first step in 
understanding the perspectives of AT 
developers toward outcomes. AT outcomes 
measurement activity has been a relatively 
new phenomenon and publications and 
discussions on AT instruments have only 
occurred in the past decade or so (Smith, 
Rust, Lauer, & Boodey, 2004). Their AT 
outcomes historical review highlights the 
impressive increase of attention in AT 
outcomes instrumentation. In future research 
it would be beneficial to specifically target 
disability manufacturers to attempt to 
improve response rate. Additionally, using the 
same survey methods for both groups would 
allow for reliable comparisons between the 
two groups. Finally, the data clearly point to 
the need to target research and development 
of AT outcomes instruments to and for 
product developers. 
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Filling out this survey indicates that I am at least eighteen years old and I am giving my informed 
consent to be a participant in this study. 





roduct Ucveloper AT Outcomes Survey 


1. Please list up to 3 products you have recently developed or have in process. 


2. The following are outcome dimensions of assistive technology. Please rate how important you 
believe each one of these dimensions is for product outcomes. 



Not at all 
Important 


Somewhat 

Important 


Extremely 

Important 

Change in Performance or 
Function 

I 

2 

3 

4 

5 

Cost 

I 

2 

3 

4 

5 

Usage: Why or why not 
used 

I 

2 

3 

4 

5 

Consumer Satisfaction 

I 

2 

3 

4 

5 

Increased Life 
Participation 

I 

2 

3 

4 

5 

Improved Quality of Life 

I 

2 

3 

4 

5 

Clinical Result/Goal 
Achievement 

I 

2 

3 

4 

5 
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Categories from ATOMS Service Directors Focus Group, April 2002 


3. How frequently have you used any of the following strategies to measure outcome during your 
process of product development? 


Not at all 

Half of 
the time 


All of the 
time 

Testing in lab by developers 1 

2 

3 

4 

5 

Focus groups of providers 1 

2 

3 

4 

5 

Focus groups of persons with ^ 

disabilities 

2 

3 

4 

5 

User usability testing in lab 1 

2 

3 

4 

5 

General field testing soliciting 
feedback from persons with 1 

disabilities 

2 

3 

4 

5 

General filed testing soliciting 
feedback from providers 

2 

3 

4 

5 

Formal research: Single subject ^ 

design 

2 

3 

4 

5 

Formal research: Group comparison ^ 
design 

2 

3 

4 

5 

Other: 1 

2 

3 

4 

5 

1 

2 

3 

4 

5 







4. If you have used formal instrumentation as a form of quantifying outcome during product 
development, how frequently have you used 



Not at all 


Half of 
the time 


All of the 
time 

a standardized, valid measure of 
functional status? 

1 

2 

3 

4 

5 

a “homemade” or adapted measure 
of improved functional performance? 

1 

2 

3 

4 

5 

a measure of client satisfaction? 

1 

2 

3 

4 

5 
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a measure of cost? 

1 

2 

3 

4 

5 

other? 

1 

2 

3 

4 

5 


5. If standardized instrumentation were available for each of the following outcomes, how 
appropriate would each of the following be for your product development? 



Wouldn’t 


Frequentl 


Always 

Product self-satisfaction 
measure 

1 

2 

3 

4 

5 

Cost measure (device, 
acquisition, fitting, learning) 

1 

2 

3 

4 

5 

Functional performance 
impact measure 

1 

2 

3 

4 

5 

Focus group protocol and 
group survey measure 

1 

2 

3 

4 

5 

Mail survey measure 

1 

2 

3 

4 

5 

Telephone survey measure 

1 

2 

3 

4 

5 

Other: 

1 

2 

3 

4 

5 

Other: 

1 

2 

3 

4 

5 

f you had valid outcome data about your products, how likely would you be to use it for th 
owing business purposes? 


Wouldn’t 


Frequentl 


Always 

General marketing 
information 

1 

2 

3 

4 

5 

Product brochures 

1 

2 

3 

4 

5 

Product revision/improvement 
during development 

1 

2 

3 

4 

5 

Identification of need for new 
products 

1 

2 

3 

4 

5 

Obtaining grant funding 

1 

2 

3 

4 

5 

Obtaining investors 

1 

2 

3 

4 

5 
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Strategies for further product ^ 
development 

2 

3 

4 

5 

Other: ^ 

2 

3 

4 

5 


Thank You! 

Please use the enclosed postage-paid envelope to return your completed survey. 
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